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ABSTRACT 

1 * 

| * • I * 

The sensitivity of the Tdrrince Test s of Creative Thinking 
with regard to an experiment e r-induced response set was 
investigated. 198 Subjects were divided into four groups* 
Before performing the Unus u al U s *s Activity CVerbal For« each 
group was given a unioue set of instruct ions • Group I received 
standard instructions and acted as \Cont rol • Other Treatments 
b er e varied in te r as o,K_the__. types of responses that were 
encouraged. 6ro u o II was encouraq e d to -be, "practical «nd 
reasonable**; Group III was encouraged to list "as aany ideas** as 
possible; and G r «up IV w*s encouraqed to include all **unusual t 
u eird t or ill 0 gicai** ideas. A multivariate Dunnett test resulted 
in Grouos III and IV beino significantly different froa the 
control (d<»01 • p<.05>. Discriminant analysis yeilded tuo 
significant functions <P<.001; P<.026>. suggesting that 
univariate analysis of the Torrance scales* of Fluencv t 
flexibility! and Or ig inal ity /4an be Misleading. The Torrance 
Tests were judqed to be highly sensitive to enper iaenter bias* 



A METH0D0L06 IC AL STUO^ OF THE TORRANCE TESTS OF CREATIVITY: 

CAN CREATIVITY BE FAKED? 

Joseph l. UUlhoft S Robert W. Lissitz 
University of Maryland 



The assessment of non-cogni t ive traits of the Individual has 
always been a concern in psychology and education* Interest 
uuring the 60 's, and 70's in affective educational proarais, and 
current interest in indentif icat ion of the talented and gifted 
have resulted in the development of many instruments for 
measuring non-cognitive abilities* Among these measures are a 
number of tests of creativity (Guilford (1956), Getzels and 
Jackson (1962)9 Wedni ck ( 1962) 9 Torrance (1966b) 9 wallach and 
Kogan (1965))* l n this study w e investigated the most commonly 
used of these tests, the Torrance Tests of Creative Thinking 
(Torrance* 1966)* v \ v 

Guilford (I9S5) developed a formulation of creativity in 
ai st inguishing between convergent and div^nfe>\t thinking* 
According to his structure of the intellect*, divergent processes 
had four dimensions: fluency, flexibility, originality, and 
elaboration* During the late 5Q's and into the mid-60's E • Paul 
Torrance developed a standardized test to measure creative 
thinking along these dimensions* In 1966 the research edit ion of 
the Torrance Tests of Creative Thinking (TTCT) was published* 
^inc e that time the TTCT has become'the most widely used battery 
of tests of creativity* Of all the instruments contained in the 
Eighth Mental Measurements Yearbook (Buros, 1978)* the TTCT 
ranked t6enty-fourth in total number of references, and had the 
most references of all tests of creativity* t 
Torrance ( 1966b) *hoe rat ionatizes creativity as : 

process cR} becgminp sensitive. to problems. 



"a process ohj oecgminp sensitive 10 orooiems, 
def ici encies , gaps in k now le doe • mi ssino e lement sr 
disharmonies, and so on: identifying the difficulty; 
seaching for solutions , making guesses, or formulating 
hypotheses about the difficulties; testing and retestiiig 
these hypotheses and possibly modifying and retesting 
them; and finally communicating the results." 
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In supporting this -definition* Torrance (1966b) states that he 
views creativity as a "generalised constellation of intellectual 
abilities, personality variables, and orob lea-sol vino traits.* 
implicit JHi this view fs the notion that the Torrance Tem | ts 
ieasure / fairly stable capacity ofyihe individual* 

row ever « t here is considerable evidence that per fonance on 
bf creativity can be Influenced by conditions of test ino f 
XlVlnd t Detolinger and Adler (197g) found that Scores on the 
•allach and K'ogan Creativity Battery (1965) were affected by 
whether or not children were interrupted fro« interesting or 
uninteresting classrooi activities inj order to be tested* They 
identified this as a "content eUecti. Kirkland (1974) and 
KfrkUndt Kilpatrlck and Barker (15^%) repl*< cat ed these flndinos 
using thfe Tor r ance Tests* Ipr^a siailar study* Boerssa and 
o'Bryan (1968) found thart sub Je.ct s* oi v en %he TTCT In t^e sap e 
room where they had been given an intelligence test on the 
previous day scored significantly lower than subjects given the 
TTCT in a dissimilar setting* 

The findings of motivational effects on TTCT performance are 
more equivocal* While Halpin and Halpin (1973) were able to 
improve TTCT sco r es under conditions of increased notivation* 
Co** Nash and Ash (1976) found no irNtect w hen TTCT performance 
was linked to rewards* C ^ f 

In addition to antecedant conditions and aotlvational 
effects* soie researchers have focused on the« instructions that 
0 re given to test takers* Most* of this r^earch has been United 
to comparing 1 standard instructions with anxiety-producing 
in st rue t Ions • Baaber. Jose and Bo ice (1975) found that 
"differential instructions (as a test* as a oi»e, without 
inst rue t ions )•• *did not affect creativity" a»ono undergraduate 
subjects* although such effects had been found with children 
(Bambe-r* 1973). Similarly* Treqthaa (1972)* us4no only the 
Originality scores of the TTCT* found no difference between 

subjects given regular instructions and subjects given 

• » 

dnxiet y -Droduci ng (1*e** ach ieveaent -or iented) Instructions* 

> 

The present study was designed to see If TTCT creativity 
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scores could be affected by subjects* perceptions of the criteria 
for Successful performance* We hypothesl fed t hat by altering the 
standard instrucflons (to emphasl ze dif ferent types of responses 
as bfina acceptable) we wduld be able to affect creativity 
^iores. The analogy we used was to *ask ourselves what effect a 
simitar experiment mioht have on a measure of academic/ 
achievement* For example, could one expect a difference in , say, 
mathematics achievement scores as a result of subjects beinq told 
that they should demonstrate extra mat bemat 1 cs abi 1 1 1 v? We felt 
that no such differences could be anticipated on a valid ' 
achievement test* However, as Crock enburo (1972) pointed out in 
her review of tests of creat i vi t y , M (w)h 1 le we cannot read i ly 
dismiss the Torr a nce Test, neither is it reasonable to conclude 

that it is a v alid measure of t^e creative process"* If our 

/ 

hypothesis were supported, we felt this would be fur t he r evidence 

i 

aqainst the validity of the TTCy, and would support the* notion 
that a major component of creativity scores is the response-set 
under which test takers operate* 
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PROCEDURE 

-j 98 subjects participated in this study* All subjects were 
qraduate students in the colleae of Education at the University 
of. Maryland and were enrolled in either a beainning or an 
intermediate statistics class. yhe nine sections of these two 
classes were randomly ass igned to or*ie of the four treatments* 
This resulted, in the, following cell distribution: Treatment 
I — two sect Ions , 49 subject s; Treatment IX--t*o sect Ions , 49 
subjects; Treatment III --two sections, 46 subjects; Treatment 
IV — three sections, 54 subjects. 

The instrument used in this study was Activity Five (Unusual 
Uses : Ca rdboard Boies) of Verbal Fdri) A of the TTCT* In this 
activity subjects are qiven fo minutes to list as many new and 
unusual uses for cardboard boxes jas they can think of* 
Performance 1n|this activity is scored on three scales: Fluency* 
Flexibility, and Originality* Fluency is defined as: "the test 



taker's ability to produce a tarqe number of Ideas wit\ word^*" 

flexibility is defined as: M a person's ability to produce a 

variety of kinds of icJeas* to shift from one approach to another, 

or ^ to use a variety of st rat eo ies Oriqinali^y is defined as: 

"the subject's ability to produce ideas that, are away .from the 

obvious* com»onplace f banal* or established;" (Torrance, 1966) 

Under typical 'administration procedures of the TTCT* orior 

to opening their test , booklets subjects are' read a set of 

instructions about how to perform in Qeneral t ; For this 

eiperUent 9 the four treatments consisted of three variants on 

these instruct ionk plus the standard set * Inserted in each test 

booklet was a copy of the treatment instructions* In addition to 

these instructions f all test booklets had a set of directions 

p r 1nted at t he top of the Paoe that were specific to th« Unusual 

Uses Activity* ^fte« opening t he *r t) 00 klets* sublets read along 

ds the experimenter read both sets (treatment instructions and 

activity instructions) aloud* The "activity instructions" th*t 

were comnon to all, treatments were: 

"Most* people throw their empty cardboard boses away* but 
they have thousands of Interesting and uKusua I. uses • In 1 
the spaces below and on the next paae 9 as ?? n y of** 

these in teres t ing and unusua I uses a ryou can think of* 
Do not limit yourself to any one-size ofc bo* . You may 
use ds many bones as you like* Do /tot limit yourself to 
the uses you have < se*n e^r heard about; think about a^s 
many new, use s as you can •*• 

The unique instructions oiven ^to Treatment I were the 

standard Instructions tyolcal ly alven to tjssr takers as a auide 

to performance on the TTCT (Torrance* 1966a)* This treatment was 

viewed a? a Control group* This arouq's Instructions were: 

••The attivlty *n this booklet will qlve vou a chaWe to 
use your imagination in thinkinq up Ideas and putting 
them Into words* There are no "right" or "wrono 1 ^ 
answers like there are in most thinas that w<e do* We 
* want you to see how many ideas you can think of and we 

thVnk • you will find this fun* Try to think of 
interesting* unusual* and clever i de asr-somet h inq that 
no one else will think of." f 

The above set^ of instructions contains four sentences* An 

analysis of the content of these sentences wa& usfd to design the 

three variant instruction sets for Treatments II* III* a n d IV* 

The first sentence was viewed as a general Introduction to the 

task and was essentially left intact In the three variants* The 



ft. 



second sentence ^as viewed as encouraalnQ test tbkers to be 

unrestricted in their responses* We interorett^d the' third 

sentence as placing importance on the nuiber af responses and as 

encouraqinq enjoyment* We viewed tHe fourth sentence Is somewhat 

restrictive, by virtue of its emphasis on the u n i|aueness of 

responses • % I 

Treatment n* the first variant, vas desiqnep to be 

considerably more restrictive than the standard sVt • Thtfse 

instructions were: \^ 

"The activity in this booklet will aive you a chance to 
think of ideas and put them in,to words* we urqe you to 
concern yc/ur se If with the p*act icati t y « and 
reasonableness of your ideas* 19 

|iere 9 the first sentence deleted the reference <■ to one's 

inaoinat ion 9 Also, delated were references both to the number of 

ideas and t^enfoyinq the lask # The "uniaueness" restriction in 

t he sta n dard instructions w' a s altered to be even more restrictive 

uy askinq test takers t o only attend to "ora c tical aftd 

reasonable" ideas* 

Treatment lilt the second variant* was desiqned to emphasize 

the "number of id e as" d spect of the standard instructions* This 

se t was : 

"The activity°in this booklet will qive you a chance to 
use your imaqination in thinkinq up ideas and puttirtq 
them into words* We want you to see now many ideas* you 
can think of for this activity*" \y \' 

T^e last v a riant« Treatment IV* was desioned to be an 

expanded version of the standard instructions that would more 

strongly urq? people to include unique ideas*, This set read as 



follows: 



"The activity in this booklet will qive you a chance to 
use your imaqination in thinkinq up ideas and puttinq 
them into words* We have found it helpful to encouraoe 
people to use their imaqination while doinq this 
activity* Remember* this is a tesr of creativity and 
that means there are no "riqht" or "wronq" answers. 
Feel free to include ALL your ideas* no matter how 
impractical or silly they may seem to you* 

"Our experience has been that whether or not you 
ARE creative is not as important as whether or not you 
THINK YOU ARE creative* 1 Thus* we stronqly urqe you to 
adopt a posture of beinq creative • to think of yourself 
as a creative person* to imaoine how a creative person 
might respond. Remember* we are interestted in answers 



tha* iake you look as creative as possible* so try try 
to tnink v of/ interestinq* unusual* and clever 
ideas--somet hjrttj that no one else will think of* 

8 



v V . \y " 

^Reaeaber to relait to think of yourselV as a 
creatiye person* to record all your ideas no matter how 

unusual, weird* or illogical the.y My seem* and * NAVE 

FUN!" \ - . A 0 

\ ,. ' * - ^ g 

Our e»oectaticm uis th-at Treatment II wpuld be the lowest* 

\ * , 

scoring group on the three scales of /lueneV* Flexibility* and 

ur iginal ity; th at Treatment III would be the highest on Fluency 

and Flexibility; that Treatment IV /would be the hiahest on 

Originality; and that Treatment I <Contro l) , would Show median 

performance on ail ihree scales # ' 

All ' test situation^ were ada i ni st ered by the saae 
experimenter* Th^ standard tiae limit for this activity was 
adhered to* Test booklets fere numbered or>ly, and subiects were 
told that their perforaance would retain anonyaOus ar^d would In 
no way affect their grade ir^Jhe course* \ 

Test booklets were scored by the Torrance Tests Scoring 
service* The Storing Service was inforaed neither of the nature 
of the study nor* of the coding scheme for numbering the tefct 

0 

booklets* 

: " RESULTS ^ 

Raw score means and standard deviations for the four qrouos, 
on each of the three scales are presented In Table I* Because 
the TT ^ T Noras-Technical Manual CTorrante.' 1966b) " recoaaend ( s^ 
tha* us^rs base the* r interpretations on the. thVee • »^ve rbal 
scores'* (p* 72)* and because our hypotheses ab, 0 ut *h e r e latjve 
performance of the four groups were based on these scales* three 



insert Tables I & II about here 

univariate F-tests were perforaed to test for differences among 
the four groups* These results are shown In 'Table II* 
inspection of Table, I indicates that our hypotheses were 
generally borne Out* The n aior dfscreDency was that Treatment I 
(Control) perforaed lowest on all .three scales* Aside froa this* 
however, Treataent II was the lowest of the three variant groups 
on each of the three scales; Treataent III was the highest of all 
groups on Fluency and Flexibility; and Treataent iv was the 
highest on Originality* The ANQVA F-tests indicated that 
significant <j roup differences occurred on all the .scales. 
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However* the anomalous performance of Treatment I was ouzzlina. 

yhy* for example* dt cf this qro u p jc 0 rc lower on Fluency than the 

■ore restricted Treatment II? *n atteaot to feetter understand 

this phenomenon and the high w ith in-ce Us correlat ions between 

lh } ie scales (see Table ill) led us to a multivariate analysis of 

— ... . ^ 

insert Jable III about here 

the data. The considerable o v e rlappinq G f the traits being 
measured* as evidenced by the high intercorrelat ions amonq the 
scales* suggested that there were ^ot three distinct dimensions 
to . TTCT performance. Hotellinq's T-squared statistic was 
computed for contrasts comparing the three variant treat«^ts 
with the Control treatment. Critical values were obtained using 
a multivariate extension of the univariate Ounnett test* which 
simultaneously <fcontrol\s for Type I error and is* consequently! 
arore powerful than the F-test usually reported with T-saoared 
statistics (Hiqazi and Dayton* 1982). The I II vs. I contract 
resulted in a T-square-16 .8 3 (p<.01>;> for ly vs. I* 
T-square*^. 44 (p<.Q5); and for II vs. I* T -s qua re =7 . 28 (P>.05). 
Given .that there w ere s ig n iiicant differences between two of the 
three variant treatments atod the Control • the Question of the 
nature of these differences arqse. A discriminant analyses was 
performed and resul ted In two ,s ighi f leant functions (o<V001 and 
p<.0?6) and one non-significant function (p<.625>. The 
correlation^ between the two significant functions and the 
original TTCT scales are sho w n in Table IV. These correlations 



insert Table IV & F iqu re 1 about here 

suggest that the first function is primarily composed of 
Flexibility and Fluency with a sizable contribution from 
uriqinality. Recalling how the TTCT scale* are defined. this 
f unci ion appears to measure how many dif f erent ideas one has and 
the number of ideas onehas.^JThe high correlation of originality 
with both Fluency and Flexibility (Table III) suoqests that that 
portion of v Originality that correlates with Fluency and 
Flexibility 1$ also part of. this first function. The second 
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function is most hiqh ly 'correlated with Oriainality. and nearly 

uncorrclated with both Flexibility an0 Fluency* Remembering how 

Originality is defined* this function seen to distinguish 

between those who have^conon *nd banal tfjeas and those who .have 

r — 
unique and unusual ideas* ouite apart from the number of* or 

differences between. <*$^s responses. Plots ofithe group ' «ian$ 

in the 2-space of t he lllscr iminant " variabl es\ are oresented in 

Figure 1. llere* It can be seen tttat the mijor difference bet ween 

treatment m and control is determined by t he f*i r $t function. 

Kecaliino that group III was uroed to come up with as many icjeas 

^ they could, this difference takes sense. Treatment Iv differs 

(roi Control on both the first function, (though not by as Much as 

does T reatment III) and the second function* T&i s was the oroup 

that was urged to record all their ideas . .no matter how illogical 

they seeded. It appears that these instructions resulted in 

performance that" yielded more ideas andt tt Jhe same time* more 

unique ideas than did the Cont ro4$ 4ns tfuct ions . Treatment II 

aoes not appear. Jo be very, distinct from the Control in this 

i-spacf, and this i$ supported by our finding no significant 

aifference here* This group's variant in&t rue t i on s j we re a 

recommendation to think of ideas that were/ practical and 

reasonable. It can be argued that 'this set of restrictions is 

not unlike the standard instructions which place restriction on 

responses ("Try to think of interesting. unusua| and clever 

ideaS~-somet hing that ho one else will think of"). 

In fact* the number of restrictions that were put upon 

Appropriate responses may have, been a N cr1tical factor in this 

study. Group III was Given the le a st restrictive instructions. 

resulting in the highest (overall) scores. Treatment IV* while 

encouraged to include a^l idea*, was restricted by directions to 

"think of interesting, unusual and clever ideas". This oroup 

scored the next most creative of the four. The standard 

instructions (Treatment I) had the fame rest ri ctions as Treatment 

IV*, yet did not jtfave the encouragement th^t the latter group fiad. 

Lack of a *?Kgni f i cant difference between Treatments I and II 

supports the observation that both of these low~scorino groups 



were similWly restricted* 

t.* 

» «■ 

" DISCUSSION ' . 

Me are aware that this study has some Imitations and that 
its- results should be viewed in Uqht of them,* First • the ent.lre 
TTCT battery of te.sts was not useJ; we only used Activity 5 of 
the Verbal fori A» While one mioht suspect that the results 
would generalize to the entire battery, further study uslno a 1 1 
the activities Is called for* Second* the Control aroup for th^l 
study was not a true control In that* \\% treatment deviated from 
standard administration practices fo.r the TTCT • Sub j e c t s In this 
study were Qlven copies of the test instructions w hile standard 
practice is to have the Inst rue t ions read to - the*, Aaain* one 
sight suspect that havlna the experimenter read the instructions 
would allow for to.re expectancy effects than providina subjects 
with their own instructions would* Finally, Intact a roups were 
randomly assigned and degrees of freedom were based upon an 
assumption of random as&iqnment Pf subjects to a^oupst Analysis 
of test scores covering course material („Gui?2*St midterms* 
finals, projects) revealed no academic differences betwee n the 
qroupst and w e can think 0 * no co«o>ll1 n a reason to support the 
notion that the groups had pre-e* ist'ina differences on Creativity 
dimensions* Keeping these points In mind, we suaaeit the 
follbwinQ conclusions* 

First* two of thf three variant treatment orouos .differed 
significantly from t^e Control aroup* This suggests that 
performance on the TTCT i^s eatrememly sensitive to the response 
set under which test takers operatf* It would be patently absurd 
to claim that our instructions to the croups actually altered 
their basic ability to be creative* Our data suooest that what 
test takers think is being asked for will have a profound effect' 
on how they will Perform* Secondly* this problem is confounded 
by the fact that ao$t applications of the TTCT are not "blind* 
studies* and that in very many cases the test administrator is 
well aware of the results that a r e desired* Thus* st rona' effects 



due both to the experimenter expectancies ... and to subjects' 
rjespc^nse sets cannot be' ruled out • . * 

.Third* there do not appear to be three distinct ereati v itjy 
traits measured by the 7 TTCT • Our data suggest that' the'r'e- are 
only two dimensions to creativity; ,Cicire|lf 11964* and Lono and 
H e n de r son ,11.944) have also reported hiqh correlations between the 
tTCT scates^of Fluency* Ftexibi li'ty and Originality (ranging from 
•60 tb •80)*; ye aqres w i t h Tho rndik e 4 * s ( 197 2) observation that 
••the evidence of consistently diffe rent meaning for the fl u ency# 
flexibility f a.n8 originality scores is almost variishino*ly small* 1 , 
and feel th^t a univariate approach to data analysis can be 

misleading* Ue recommend that researchers be v*ry cautious when 

w • ■ * *\ m • I--, 

in te rpret-t ino . TTCT scale scores as being measures of the traits 

as djefined in the test manual • . 1 

Our first point seems to be the most critical one. The 

degree to which test takers- fee I res tricted or encouraged on the 

responses ipay well have a critical, effect on their performance* 

-That our 'modest ^treatments were able to induce significant 

differences in creativity performance is' puite striking* One way 

to i9t e rpret these differences is/ to view our treatments as 

••mini" training programs on how to be creative. Our data suggest 

that reseachers* interested in evaluating the effectiveness of 

creativity training cannot us e a\baS? r ^ ne " no thanae" as ^a 

null-criterion, aur findings indicate that the M n Q ^difference" 

condition for fheTo r ^a n ce Te&ts^may have a" irihererft-. variability 

of at least! orie standard deviation (see Table I). It is known 

that experimental subjects may react to an' experimenter's 

expectations (Orne f 1969; Rosenthal. 196.9) • Our f indicia of t he 

extreme sensitivity of the TTCT to experiment errinduced/ response 

sets leads us to conclude that ev^n under conditions ^riArous 

cont rol, studies using the Torrance Tests should be vi e wed ^ wi th 

extreme caution. Mot only may the results be a methodological 

artifact, but the interpretation of creativity as a w real H 

internal characteristic of a person is subject to suspicion. 
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Fn. 1 I 
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-.156 
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Table IV: Correlations between discriminant 
functions and TTCT scales. 
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Figure 1. Plots of group means on w 

discriminant functions 
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'* TTCT Scale Mean Square Error F (3.194) p less than 

Fluency 82.270 5.602 .001 

Flexibility 10.933 6.672 .001 

Originality 151.625 • 5.068 .002 

Tab 1^ II: Univariate F-tests for" group" differences 

on each of the 3 scales. 



T-tf 



i 



FLUENCY FLEXIBILITY ORIGINALITY 
Treatment I MEAN: 22.53 U.63 23.76 

, n ■ kS S.D.: , 8.06 ' • 3.'46 *' 10.90 

Treatment II . MEAN: ' 24.00 12.^5 * |< 28.20 

n V %9 S.D.: 7.28 ! 2.78 .12.18 

Treatment III MEAN: 29.22 14.35 31.09 

n •■ h6 S.D.: 9.78. 3.42 11.05 

Treatment IV MEAN: 27.54 13. 87 32.67 

. , n = 54 S.D.: " \) 10.64 3.48 '1*1.47 

■ :• / 

Table I: Means and standard deviations for the k groups 

On each of the 3 TTCT scales. 

- — V X. — : • 



Flu Flex Prig 
Fluency 1.000 

Flexibility .687 1.000 

Originality .838 .749 1.000 

Table III: Vi thin-cells correlations among 
» ■ the scales. 



